Papers with ensemble methods
Mimic and Conquer: Heterogeneous Tree Structure Distillation for Syntactic NLP (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing work on integrating syntactic information into neural networks uses a single tree, such as a constituency or a dependency tree. |
| Approach: | They propose a method to integrate heterogeneous structure knowledge into a unified sequential LSTM encoder. |
| Outcome: | The proposed method outperforms tree encoders on four syntax-dependent tasks and is efficient and accurate. |
How Can We Know What Language Models Know? (2020.tacl-1)
Copied to clipboard
| Challenge: | Recent work examines knowledge contained in language models by having the LM fill in the blanks of prompts such as “Obama is a __ by profession”. |
| Approach: | They propose mining-based and paraphrasing-based methods to automatically generate high-quality and diverse prompts, as well as ensemble methods to combine answers from different prompts. |
| Outcome: | The proposed methods improve accuracy from 31.1% to 39.6% on the LAMA benchmark for extracting relational knowledge from LMs. |
Bag of Tricks for In-Distribution Calibration of Pretrained Transformers (2023.findings-eacl)
Copied to clipboard
| Challenge: | Recent studies show that pre-trained language models (PLMs) often predict over-confidently. |
| Approach: | They propose to use ensemble learning and data augmentation to improve confidence calibration for PLMs by combining calibration techniques with a trade-off between accuracy and classification. |
| Outcome: | The proposed calibration method improves classification accuracy and confidence in pre-trained language models by combining several calibration techniques. |
Aggregation of Reasoning: A Hierarchical Framework for Enhancing Answer Selection in Large Language Models (2024.lrec-main)
Copied to clipboard
Zhangyue Yin, Qiushi Sun, Qipeng Guo, Zhiyuan Zeng, Xiaonan Li, Tianxiang Sun, Cheng Chang, Qinyuan Cheng, Ding Wang, Xiaofeng Mou, Xipeng Qiu, Xuanjing Huang
| Challenge: | Recent advances in Chain-of-Thought prompting have facilitated significant breakthroughs for Large Language Models (LLMs) in complex reasoning tasks. |
| Approach: | They propose a hierarchical reasoning aggregation framework to address this problem . they propose dynamic sampling to adjust the number of reasoning chains . |
| Outcome: | The proposed framework outperforms existing ensemble methods on complex reasoning tasks. |
Routing to the Expert: Efficient Reward-guided Ensemble of Large Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing ensemble methods for Large Language Models focus on reward model ranking of outputs, leading to significant computation overhead. |
| Approach: | They propose a reward-guided routing method distilling rewards on training queries to train a routing function. |
| Outcome: | The proposed method outperforms the best single model and ranks first on 44% of tasks. |
Dynamic Classification in Web Archiving Collections (2020.lrec-1)
Copied to clipboard
| Challenge: | a growing number of research libraries, museums, and archives are embracing Web Archiving as a mechanism to collect born-digital material made available via the Web. |
| Approach: | They propose to use dynamic fusion models to find the model that performs best on a variety of document types. |
| Outcome: | The proposed model outperforms individual models and other ensemble methods on three datasets. |
Harnessing Consistency for Robust Test-Time LLM Ensemble (2026.findings-eacl)
Copied to clipboard
Zhichen Zeng, Qi Yu, Xiao Lin, Ruizhong Qiu, Xuying Ning, Tianxin Wei, Yuchen Yan, Jingrui He, Hanghang Tong
| Challenge: | Existing efforts to improve LLM ensemble quality have focused on model consistency, but failures are often due to heterogeneous tokenization schemes and varying model expertise. |
| Approach: | They propose a plug-and-play technique that harnesses model consistency for robust LLM ensemble. |
| Outcome: | The proposed technique improves ensemble performance and robustness against erroneous signals. |
Devil’s Advocate: Novel Boosting Ensemble Method from Psychological Findings for Text Classification (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing ensemble methods that combine submodels to create a composite model can improve model performance by diminishing model bias and variance. |
| Approach: | They propose a method which uses a deliberately dissenting model to force other submodels within the ensemble to better collaborate. |
| Outcome: | The proposed method shows comparable or improved performance on 5 text classification tasks when compared to conventional methods. |
Character-level Language Models for Abbreviation and Long-form Detection (2024.lrec-main)
Copied to clipboard
| Challenge: | Abbreviations and long forms are textual elements that are present in scientific communication . non-recognition of abbreviation and long form can lead to a negative impact on information retrieval . |
| Approach: | They propose to train and test language models for automatically identifying abbreviations and long forms . they use existing datasets annotated with abbrevations and their associated long forms to test them . |
| Outcome: | The proposed model can detect abbreviations and long forms on biomedical data . the proposed model improves on a previously untested dataset with biomedically-annotated datasets . |
DREAM: Deployment of Recombination and Ensembles in Argument Mining (2023.emnlp-main)
Copied to clipboard
| Challenge: | Current approaches to Argument Mining (AM) take a holistic view of the overall pipeline. |
| Approach: | They propose a framework that allows for the (automated) combination of AM components instead of all-new solutions. |
| Outcome: | The proposed framework outperforms the best single systems in terms of accuracy measured by an AM benchmark. |
Bridging the Gap between Different Vocabularies for LLM Ensemble (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing methods to combine large language models with complete outputs have limited effectiveness . lexical gaps between different LLMs hinder dynamic correction and enhancement . |
| Approach: | They propose a method to ensemble large language models via Vocabulary Alignment (EVA) they learn mappings between vocabularies of different LLMs with overlapping tokens . |
| Outcome: | The proposed method bridges the lexical gap among various LLMs, enabling meticulous ensemble at each generation step. |
DMoERM: Recipes of Mixture-of-Experts for Effective Reward Modeling (2024.findings-acl)
Copied to clipboard
| Challenge: | Using a reward model (RM) to improve the effectiveness of large language models, there are two challenges in training. |
| Approach: | They propose a reward model (RM) that is a proxy of human preferences and assigns scores to the outputs of the large language model (LLM) a human annotation consistency rate of 60% to 75% is causing training data to contain a lot of noise. |
| Outcome: | The proposed model outperforms state-of-the-art ensemble methods and mitigates the overoptimization problem. |
Frustratingly Easy Model Ensemble for Abstractive Summarization (D18-1)
Copied to clipboard
| Challenge: | Existing studies on compressing or distilling ensemble models have shown that they increase computational costs and reduce performance. |
| Approach: | They propose an unsupervised method that combines multiple models by selecting a majority-like output in post-processing. |
| Outcome: | The proposed method performs better than the current ensemble methods on a news-headline-generation task. |
Revisiting the Practical Effectiveness of Constituency Parse Extraction from Pre-trained Language Models (2022.coling-1)
Copied to clipboard
| Challenge: | Constituency Parse Extraction from Pre-trained Language Models (CPE-PLM) is a new paradigm that attempts to induce constituency parse trees based on the internal knowledge of pre-tried language models. |
| Approach: | They propose to use constituency parse trees from pre-trained language models to induce constituency trees by introducing a set of heterogeneous PLMs combined using two advanced ensemble methods. |
| Outcome: | The proposed approach is more effective than typical supervised parsers in few-shot settings. |
Ensembling Large Language Models with Process Reward-Guided Tree Search for Better Complex Reasoning (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for ensembling language models fail to address complex reasoning tasks. |
| Approach: | They propose a framework for process-level ensembling of large language models using Monte Carlo tree search. |
| Outcome: | The proposed framework outperforms both language model decoding and language model ensemble methods on five reasoning benchmarks. |
Cool-Fusion: Fuse Large Language Models without Training (2025.acl-long)
Copied to clipboard
| Challenge: | Cool-Fusion is a simple yet effective approach to combine two or more heterogeneous large language models . |
| Approach: | They propose a method that fuses the knowledge of two or more heterogeneous large language models to leverage complementary strengths. |
| Outcome: | The proposed method increases accuracy from three strong source LLMs on GSM8K by 17.4%. |
Hit the Sweet Spot! Span-Level Ensemble for Large Language Models (2025.coling-main)
Copied to clipboard
| Challenge: | a recent study focused on sample-level and token-level ensembles, which hinder dynamic correction and enhancement of outputs during the generation process. |
| Approach: | They propose a span-level ensemble method that balances real-time adjustments and accurate ensemble decisions. |
| Outcome: | The proposed method improves performance across language generation tasks significantly. |
Measuring What Matters: Evaluating Ensemble LLMs with Label Refinement in Inductive Coding (2025.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) are prone to inconsistencies and individual biases, limiting their reliability. |
| Approach: | They propose a framework that combines ensemble methods with code refinement methodology to address these challenges. |
| Outcome: | The proposed framework outperforms large language models and LLMs with a low-rank averaging and a moderator-based mechanism to simulate human consensus. |
Two Birds One Stone: Dynamic Ensemble for OOD Intent Classification (2023.acl-long)
Copied to clipboard
| Challenge: | Out-of-domain (OOD) intent classification is an active field of natural language understanding . previous studies have suggested that PTMs would be "overthinking" the semantic features of the sample in the open-world scenario . |
| Approach: | They propose a method that allows the model to decide whether to make a decision on OOD classification early during inference. |
| Outcome: | The proposed method can improve inference speed and achieve significant performance improvements. |
RLAE: Reinforcement Learning-Assisted Ensemble for LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing ensemble methods for ensembling large language models rely on fixed weighting strategies that fail to adapt to dynamic, context-dependent characteristics of LLMs. |
| Approach: | They propose a framework that reformulates LLM ensemble through a Markov Decision Process. |
| Outcome: | The proposed framework outperforms existing methods by 3.3% on a diverse set of tasks while achieving lower time latency. |
MoSE: Modality Split and Ensemble for Multimodal Knowledge Graph Completion (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to predict missing entities share relation representation across modalities, which results in mutual interference between modality. |
| Approach: | They propose a framework for multimodal knowledge graph completion that learns modality-split relation embeddings for each modality instead of a single modality shared one. |
| Outcome: | The proposed framework outperforms state-of-the-art methods on three KG datasets. |
Improving Multi-view Document Clustering: Leveraging Multi-structure Processor and Hybrid Ensemble Clustering Module (2024.lrec-main)
Copied to clipboard
| Challenge: | Experimental results show that DMsECN outperforms existing models for document clustering . |
| Approach: | They propose a multi-view document clustering model with a processor and hybrid module . they demonstrate that DMsECN outperforms existing models by creating a consensus structure from multiple clustering structures. |
| Outcome: | The proposed model outperforms existing models on four multi-view document clustering datasets. |
Two-Stage Fine-Tuning for Improved Bias and Variance for Large Pretrained Language Models (2023.acl-long)
Copied to clipboard
| Challenge: | Recent work challenges the bias-variance trade-off . large pretrained models can have large variance and overfit domain-specific data . |
| Approach: | They propose a bias-variance trade-off that implies learning methods need to balance complexity with data size to minimize under-fitting and over-fit. |
| Outcome: | The proposed method achieves strong results on SuperGLUE and clinical information extraction tasks. |